[FEATURE](schema) Add IntegerConstraint; fail codegen on divergent union column types#553
Draft
Seth Fitzsimmons (sethfitz) wants to merge 4 commits into
Draft
[FEATURE](schema) Add IntegerConstraint; fail codegen on divergent union column types#553Seth Fitzsimmons (sethfitz) wants to merge 4 commits into
Seth Fitzsimmons (sethfitz) wants to merge 4 commits into
Conversation
Introduce IntegerConstraint, a field constraint that requires a floating-point value be a whole number. It validates in Pydantic (rejecting non-integral floats), emits multipleOf: 1 in JSON Schema, and dispatches to a new check_integer PySpark expression. The constraint lets a count be modeled as float64 (for a uniform Parquet column) while still rejecting fractional values. Wire it through the codegen pipeline: constraint dispatch, the check_integer runtime expression, invalid-value generation for conformance tests, and markdown constraint description. Base-row synthesis needs no change -- the accompanying bounds drive the value. Refs #472 Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
… float64 VehicleAxleCountSelector.value was the only VehicleSelector arm typed uint8; the other four use float64. Flattened to a Spark struct, all five arms share one `value` column -- the codegen widened uint8 to DoubleType to make that column work, an implicit compatibility requirement nothing in the model surfaced. Change value to float64 with IntegerConstraint (must be a whole number) and a minimum of 1. This makes the union's Spark type explicit rather than implicit: `value` was already DoubleType via widening, and is now DoubleType natively. Generalizing schema_builder and union_extraction (previous commit) is a prerequisite: without it, this change fails at generation with a diverging-constraints error, since axle count's `ge=1 + integral` rule differs from the other arms' `ge=0`. Regenerate the PySpark modules. The only changes are to Segment (the sole model embedding VehicleSelector): new `bounds` (ge=1.0) and `integer` checks gated to the axle_count arm, new conformance-test scenarios exercising them, and check-function/label renumbering from the added collision in the `value` group. No other model tripped the generalized union type-divergence check. Refs #472 Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
added
the
change type - minor 🤏
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
GitHub Actions
Inactive
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
…lit-field labels by arm check_integer tested `col != F.floor(col)`. F.floor casts to LongType and saturates doubles at 2^63, so integral values like 1e30 and 9.9e18 failed the check even though Pydantic accepts them (float.is_integer() is True). Test the fractional part in double space instead (`col % 1 != 0`): those large integral doubles pass, and 2.5, NaN, and infinity fail -- matching float.is_integer(). Correct the docstring, which claimed floor "passes" NaN; floor(NaN) returns 0, so the old code rejected NaN through a false mechanism. Group field-check label collision suffixes by union arm rather than by (label, name) occurrence. A check present in only one arm -- the axle arm's integer check, absent from the dimension arms -- kept the bare `value` label while its arm siblings took `value_0`. It now shares the arm's suffix. Single-arm labels keep the per-occurrence rule, so unsplit fields and same-name bounds pairs (confidence, cartography) are unchanged. Regenerate pyspark output: the vehicle `value` integer check on access_restrictions, prohibited_transitions, and speed_limits reports `value_0`. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Arm-grouped label suffixing cannot distinguish two same-name checks landing in one arm of a split union field. No current schema produces that shape; if one ever does, generation now raises instead of emitting indistinguishable violation labels. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #472.
Implements the solution agreed in the issue thread, on top of the PySpark codegen rewrite:
IntegerConstraintfield constraint: validates integral floats in Pydantic, emitsmultipleOf: 1in JSON Schema, and dispatches to a newcheck_integerPySpark expression with generated conformance-test coverage.VehicleAxleCountSelector.valuebecomesfloat64withIntegerConstraintand a minimum of 1 (wasuint8).valuecolumn was alreadyDoubleTypevia widening; it is nowDoubleTypenatively), with new arm-gated integer/bounds checks.Union arms that share a field name and type but diverge in constraints now produce separate arm-gated checks instead of an extraction error; violation labels for a split field's checks are suffixed by arm (
value_0,value_1), and generation fails loudly if two same-name checks ever land in one arm.